Stochastic Dynamic Optimization Problem Research Articles

Regression based stochastic energy management strategy for a grid connected microgrid using Artificial Electric Field Algorithm

Recent decade has witnessed steep technological advancement in the renewable energy sector due to growing concern of climate change and emission cut target. The lack of availability of fossil fuels and the high price of the same have made the use of Renewable Energy Sources (RES) as a promising approach for generating clean and sustainable energy at local level. This persuades the implementation of Microgrid (MG) supported with RES, to cater various types of load demand. However, planning and optimizing a MG operation have become challenging due to the intermittent nature of RES and uncertain loads. To address this issue, this paper introduces a Stochastic Energy Management Strategy (SEMS) for a grid connected MG comprising Photovoltaic, Wind Turbine, Fuel Cell, Micro Turbine, Battery Energy Storage and electrical as well as heat energy demand. The volatilities of RES power output and energy demand are modeled via Gaussian Process Regression (GPR) and Monte Carlo Simulation (MCS) respectively. The problem is framed as non-linear (NL) dynamic stochastic optimization problem and solved using ‘Artificial Electric Field Algorithm (AEFA)’. Here, minimization of expected value of operational and emission cost are taken as objective functions; while various practicality aspects are modeled as constraints in the problem. Further, the efficacy of the proposed approach is validated around the first central moment of the load. Various case studies are performed to estimate the day ahead optimum operating schedule of the Distributed Energy Resources (DER) and to assess the impact of load uncertainties on DER operation and total MG operation cost. Further, the efficacy of the GPR is verified through comparison with MCS based result.

Performance analysis of machine learning based prediction models in assessing optimal operation of microgrid under uncertainty

Of late, the exponential rise in the global population is driving higher energy demand. However, the rapid depletion of conventional fossil fuels and growing environmental concerns have prompted the evolution of alternative energy sources. To this end, Microgrid (MG) with Renewable Energy Sources (RES) has emerged as popular means of small-scale localized power grid. However, planning of MG operation poses challenges due to the inherent variability and stochasticity in RES power output and energy demand. On account of this, the present study introduces a Stochastic Energy Management Strategy (SEMS) for a grid-connected MG incorporating Micro-Turbine, Fuel-Cell, RES, Battery Energy Storage, and electrical and heat energy demand. The stochasticity of RES is forecasted through a hybrid prediction model (sARIMA-GRU) and the uncertain demand is estimated via 'Monte Carlo Simulation.' The proposed problem is formulated as a dynamic non-linear stochastic optimization problem. It seeks to minimize the expected value of MG operational cost satisfying the practical constraints. Addressing this, a newly developed ‘Artificial Electric Field Algorithm (AEFA)' is utilized. Several case studies are performed to assess MG operation under varied operating conditions. Moreover, the present study analyses the impact of uncertainty on energy contribution from DER, grid dependency, and MG operation cost. Comparative analysis reveals that sARIMA-GRU outperforms other contemporary prediction models. It is noteworthy that the superior prediction accuracy of sARIMA-GRU leads to lower MG operation costs. Moreover, statistical analysis and convergence confirm the proficiency of applied AEFA over state-of-the-art Grey Wolf Optimization and Firefly Algorithm in solving the proposed problem.

Flattening Energy-Consumption Curves by Monthly Constrained Direct Load Control Contracts

California Utility Firm Implements Innovative Model, Reducing Costs by 4% A California utility firm has successfully implemented a pioneering model to balance electricity demand and supply while minimizing costs. By utilizing direct load control contracts (DLCCs), the firm can reduce energy consumption during peak hours. Researchers developed an integer stochastic dynamic optimization problem that considers monthly and annual constraints, allowing for effective execution of DLCCs. Incorporating a “reduce-to-threshold” policy to flatten energy-consumption curves during high demand, the model was verified using real data from the California Independent System Operator. When implemented, the utility firm achieved an impressive cost reduction of approximately 4%. Sensitivity analysis was conducted to enhance customer experience and improve DLCC contract features. The success of this innovative model highlights the potential of DLCCs and advanced optimization techniques in the energy sector, offering a blueprint for other utility companies seeking to optimize grid stability and reduce costs.

A Nonparametric Algorithm for Optimal Stopping Based on Robust Optimization

Optimal stopping is a fundamental class of stochastic dynamic optimization problems with numerous applications in finance and operations management. In “A nonparametric algorithm for optimal stopping based on robust optimization,” the author introduces an approach based on robust optimization for computing near-optimal solutions for computationally demanding stochastic optimal stopping problems with known probability distributions. Through this new use of robust optimization, the paper develops new algorithms for solving stochastic optimal stopping problems that are practical and can outperform state-of-the-art simulation-based algorithms in the context of pricing high-dimensional financial options.

A note on the existence of optimal stationary policies for average Markov decision processes with countable states

In many practical stochastic dynamic optimization problems with countable states, the optimal policy possesses certain structural properties. For example, the (s,S) policy in inventory control, the well-known cμ-rule and the recently discovered c/μ-rule (Xia et al. (2022)) in scheduling of queues. A presumption of such results is that an optimal stationary policy exists. There are many research works regarding to the existence of optimal stationary policies of Markov decision processes with countable state spaces (see, e.g., Bertsekas (2012); Hernández-Lerma and Lasserre (1996); Puterman (1994); Sennott (1999)). However, these conditions are usually not easy to verify in such optimization problems. In this paper, we study the optimization of long-run average of continuous-time Markov decision processes with countable state spaces. We provide an intuitive approach to prove the existence of an optimal stationary policy. The approach is simply based on compactness of the policy space, with a special designed metric, and the continuity of the long-run average in the space. Our method is capable to handle cost functions unbounded from both above and below, which makes a complementary contribution to the literature work where the cost function is unbounded from only one side. Examples are provided to illustrate the application of our main results.

Optimal stochastic inventory-distribution strategy for damaged multi-echelon humanitarian logistics network

The quick distribution of relief goods is vital in alleviating human suffering in affected areas. A relief strategy that delivers (pushes) goods to the affected population depending on the predicted demand is critical, especially during the early post-disaster period when accurate demand information is lacking. Several inventory-distribution strategies based on multi-echelon humanitarian logistics networks have been previously investigated to facilitate a quick response to demand information. However, natural disasters have raised doubts about the performance of these networks. This paper presents the disaster conditions under which conventional multi-echelon networks are conducive to push-mode strategies that deliver relief goods depending on the predicted demand. Specifically, an approximate solution to the optimal inventory-distribution strategy is analytically derived from a dynamic stochastic optimization problem using deterministic approximation and decomposition. Several numerical validations ensure the optimality and practical applicability of the strategy. The approximate strategy identifies disaster conditions that do not contribute to push-mode strategies in conventional multi-echelon networks and proposes an alternative network structure.

Pandemic portfolio choice

COVID-19 has taught us that a pandemic can significantly increase biometric risk and at the same time trigger crashes of the stock market. Taking these potential co-movements of financial and non-financial risks into account, we study the portfolio problem of an agent who is aware that a future pandemic can affect her health and personal finances. The corresponding stochastic dynamic optimization problem is complex: It is characterized by a system of Hamilton-Jacobi-Bellman equations which are coupled with optimality conditions that are only given implicitly. We prove that the agent’s value function and optimal policies are determined by the unique global solution to a system of non-linear ordinary differential equations. We show that the optimal portfolio strategy is significantly affected by the mere threat of a potential pandemic.

Influence of Inherent Characteristic of PV Plants in Risk-Based Stochastic Dynamic Substation Expansion Planning Under MILP Framework

A suitable probabilistic scenario set of load demand and natural characteristics of renewable energy is becoming a crucial issue in power system planning studies. Properly addressing the impact of potentially thousands of residential PV plants on the resilience and reliability needs of substations necessitates the representation of inherent relations between photovoltaics and the load throughout the long-term planning period. The optimal planning of substation expansions is achievable through proper modeling of input parameters which describes the characteristics of the service areas. In this paper, the co-existence of PV plants and the load in a service area under three different states such as daytime with clear-sky and no-fault, daytime with abnormal events, and nighttime are incorporated into the stochastic dynamic optimization problem by using scenario-based approach. The scenario tree of the problem is branched from three different bases simultaneously instead of only one as in conventional approach. This paper also combines the risk-constrained stochastic dynamic SEP problem and Mixed Integer Linear Programming (MILP) framework under one roof. The comparison between integrating inherent characteristics of PV plants with and without considering abnormal events into the optimization is performed to show the impact of suitable probabilistic model on dynamic nature of investment decisions.

A data-driven approach for a class of stochastic dynamic optimization problems.

Dynamic stochastic optimization models provide a powerful tool to represent sequential decision-making processes. Typically, these models use statistical predictive methods to capture the structure of the underlying stochastic process without taking into consideration estimation errors and model misspecification. In this context, we propose a data-driven prescriptive analytics framework aiming to integrate the machine learning and dynamic optimization machinery in a consistent and efficient way to build a bridge from data to decisions. The proposed framework tackles a relevant class of dynamic decision problems comprising many important practical applications. The basic building blocks of our proposed framework are: (1) a Hidden Markov Model as a predictive (machine learning) method to represent uncertainty; and (2) a distributionally robust dynamic optimization model as a prescriptive method that takes into account estimation errors associated with the predictive model and allows for control of the risk associated with decisions. Moreover, we present an evaluation framework to assess out-of-sample performance in rolling horizon schemes. A complete case study on dynamic asset allocation illustrates the proposed framework showing superior out-of-sample performance against selected benchmarks. The numerical results show the practical importance and applicability of the proposed framework since it extracts valuable information from data to obtain robustified decisions with an empirical certificate of out-of-sample performance evaluation.

Soft actor-critic –based multi-objective optimized energy conversion and management strategy for integrated energy systems with renewable energy

As an essential development direction of energy internet, integrated energy system with interdisciplinary techniques is of great significance to promote multi-energy cooperation, realize low-carbon economic operation, and improve the flexible scheduling potential. This paper presents the study of an integrated power, heat and natural-gas system consisting of energy coupling units and wind power generation interconnected via a power grid. A deep reinforcement learning –based energy scheduling strategy is proposed to optimize multiple targets, including minimizing operational costs and ensuring power supply reliability. By scheduling the output of energy units, the economy and reliability of the considered system are improved. Taken diversified uncertainties into account, like intermittent of wind power and flexibility of load demand, the stochastic dynamic optimization problem is modeled as Markov decision process, and a soft actor-critic algorithm is introduced to solve the complex scheduling problem. The optimized decision-making action can be identified by the soft actor-critic algorithm through empirical learning without prediction information and prior knowledge. In the simulation, the proposed SAC-based agent has robust performance on solving optimization problems of different scenarios. Besides, the comparison study is carried out among benchmark reinforcement learning and heuristic algorithms, parameters of which are specifically given. The results demonstrate that when optimizing the comprehensive profits, the developed strategy reduced costs by up to 21.66%, compared to other algorithms.

Pandemic Portfolio Choice

COVID-19 has taught us that a pandemic can significantly increase biometric risk and at the same time trigger crashes of the stock market. Taking these potential co-movements of financial and non-financial risks into account, we study the portfolio problem of an agent who is aware that a future pandemic can affect her health and personal finances. To model the financial and non-financial aspects of a pandemic, our framework involves a Markov chain capturing the pandemic state of the society and a state-dependent jump-diffusion model of the stock market. In particular, we allow for pandemic-induced stock market crashes similar to the crash that we witnessed at the beginning of March, 2020. The corresponding stochastic dynamic optimization problem is complex: It is characterized by a system of Hamilton-Jacobi-Bellman equations which are coupled with optimality conditions that are only implicitly given. We prove that the agent's value function and optimal policies are determined by the unique global solution to a system of non-linear ordinary differential equations. From an economic point of view, we show that the optimal portfolio strategy is significantly affected by the mere threat of a potential pandemic.

A nonparametric algorithm for optimal stopping based on robust optimization

Optimal stopping is a class of stochastic dynamic optimization problems with applications in finance and operations management. In this paper, we introduce a new method for solving stochastic optimal stopping problems with known probability distributions. First, we use simulation to construct a robust optimization problem that approximates the stochastic optimal stopping problem to any arbitrary accuracy. Second, we characterize the structure of optimal policies for the robust optimization problem, which turn out to be simple and finite-dimensional. Harnessing this characterization, we develop exact and approximation algorithms for solving the robust optimization problem, which in turn yield policies for the stochastic optimal stopping problem. Numerical experiments show that this combination of robust optimization and simulation can find policies that match, and in some cases significantly outperform, those from state-of-the-art algorithms on low-dimensional, non-Markovian optimal stopping problems from options pricing.

Optimal Grid Selection for the Numerical Solution of Dynamic Stochastic Optimization Problems

This paper unites numerical literature concerning the optimal linear approximation of convex functions with theory on the consumption savings problems of households in macro economies with idiosyncratic risk and incomplete markets. Construction of a grid for the linear approximation of household savings behavior which is optimal in the sense of minimizing the largest absolute error is characterized in a standard environment with income fluctuations and a single savings asset. For wealthy households, the grid is characterized asymptotically as having a density which decreases in household wealth. For domains which include resource poor households, the optimal grid is seen to have non-monotonic grid point density for standard parameters. This feature contradicts conventional rules for constructing grids, and is related to non-monotonic curvature in the savings function for low resource holdings. Approximate optimal grids are seen to outperform standard grid constructs according to a variety of accuracy measures at the cost of significantly increased computational time, and efficiency-improving alternatives are given.

Optimal concession contracts for oil exploitation

This paper studies the relationship between a government and private companies for the exploitation of an oilfield by means of concession-like contracts, i.e. concessions and Production Sharing Agreements. At this aim, we develop and solve a dynamic stochastic optimization problem in a real option framework. The model takes into account crucial as well as actual features of the real world, such as: the twofold goal of governments who must mediate between social interests and revenue maximization from concessions; the incentive for the private party to ”over exploit” natural resources and uncertainty over future payoffs. The results obtained can help policy makers in pursuing the delicate task of setting the ”right” terms of concession-like contracts, meaning that policy makers can have at least a benchmark to start interacting with private parties. This phase is particularly difficult for a number of reasons, such as the need to trade contrasting interests off, high risk of corruption and the fact that negotiations are made difficult by the high level of uncertainty due to incomplete or even faulty information.

The Limits of Operational Excellence in the Seed Industry Under Climate Change

This paper analyzes the optimal production decisions of a seed manufacturer confronted by the increased likelihood of extreme weather events that amplify the volatility of the yield in the seed production process. We model the process as a discrete-time stochastic dynamic optimization problem where, in each period, the manufacturer solves an optimization problem with two stages: The planting stage, where the manufacturer chooses the quantity of hybrid seeds to produce through a network of outsourced farmers, and the allocation stage, where the manufacturer allocates the resulting yield from the planting stage to different markets with varying profit margins. We operationalize the effect of climate change through an increased probability of extreme events. We prove that, as the extreme event probability increases (a) the manufacturer’s expected profits decrease while the optimal planting quantity grows; (b) the expected total yield of the planting stage decreases, reducing the profit of the farmers who work with the manufacturer; (c) the expected allocation quantity to different markets decreases, and this decrease is more significant in low-margin markets. Finally, we present a simulation model calibrated to industry data. We find that even a small increase in the likelihood of extreme weather events can dramatically change optimal production decisions and have significant negative effects on the profitability of commercial seeds, on the availability of seeds in low-margin markets, on the profits obtained by farmers involved in the seed production process and, on the value of postponement strategy.

Constrained Reinforcement Learning for Dynamic Optimization under Uncertainty

Dynamic real-time optimization (DRTO) is a challenging task due to the fact that optimal operating conditions must be computed in real time. The main bottleneck in the industrial application of DRTO is the presence of uncertainty. Many stochastic systems present the following obstacles: 1) plant-model mismatch, 2) process disturbances, 3) risks in violation of process constraints. To accommodate these difficulties, we present a constrained reinforcement learning (RL) based approach. RL naturally handles the process uncertainty by computing an optimal feedback policy. However, no state constraints can be introduced intuitively. To address this problem, we present a chance-constrained RL methodology. We use chance constraints to guarantee the probabilistic satisfaction of process constraints, which is accomplished by introducing backoffs, such that the optimal policy and backoffs are computed simultaneously. Backoffs are adjusted using the empirical cumulative distribution function to guarantee the satisfaction of a joint chance constraint. The advantage and performance of this strategy are illustrated through a stochastic dynamic bioprocess optimization problem, to produce sustainable high-value bioproducts.

Epiconvergence of relaxed stochastic optimization problems

We consider relaxation of almost sure constraint in dynamic stochastic optimization problems and their convergence. We show an epiconvergence result relying on the Kudo convergence of σ-algebras and continuity of the objective and constraint operators. We present classical constraints and objective functions with conditions ensuring their continuity. We are motivated by a Lagrangian decomposition algorithm, known as Dual Approximate Dynamic Programming, that relies on relaxation, and can also be understood as a decision rule approach in the dual.

Dynamic intertemporal utility optimization by means of Riccati transformation of Hamilton–Jacobi–Bellman equation

In this paper we investigate a dynamic stochastic portfolio optimization problem involving both the expected terminal utility and intertemporal utility maximization. We solve the problem by means of a solution to a fully nonlinear evolutionary Hamilton–Jacobi–Bellman (HJB) equation. We propose the so-called Riccati method for transformation of the fully nonlinear HJB equation into a quasi-linear parabolic equation with non-local terms involving the intertemporal utility function. As a numerical method we propose a semi-implicit scheme in time based on a finite volume approximation in the spatial variable. By analyzing an explicit traveling wave solution we show that the numerical method is of the second experimental order of convergence. As a practical application we compute optimal strategies for a portfolio investment problem motivated by market financial data of German DAX 30 Index and show the effect of considering intertemporal utility on optimal portfolio selection.

A non-linear estimation and model predictive control algorithm based on ant colony optimization

A new heuristic controller, called the continuous ant colony controller, is proposed for non-linear stochastic Gaussian/non-Gaussian systems. The new controller formulates the state estimation and the model predictive control problems as a single stochastic dynamic optimization problem, and utilizes a colony of virtual ants to find and track the best estimated state and the best control signal. For this purpose, an augmented state space is defined. An integrated cost function is also defined to evaluate the points of the augmented state space, explored by the ants. This function minimizes simultaneously the state estimation error, tracking error, control effort and control smoothness. Ants search the augmented state space dynamically in a similar scheme to the optimization algorithm, known as the continuous ant colony system. Performance of the new model predictive controller is evaluated for three non-linear problems. The problems are a non-linear continuous stirred tank reactor, a non-linear cart and spring system, and the attitude control of a non-linear quadrotor. The results verify successful performance of the proposed algorithm from both estimation and control points of view.

Expected utility maximization and conditional value-at-risk deviation-based Sharpe ratio in dynamic stochastic portfolio

In this paper we investigate the expected terminal utility maximization approach for a dynamic stochastic portfolio optimization problem. We solve it numerically by solving an evolutionary Hamilton-Jacobi-Bellman equation which is transformed by means of the Riccati transformation. We examine the dependence of the results on the shape of a chosen utility function in regard to the associated risk aversion level. We define the Conditional value-at-risk deviation ($CVaRD$) based Sharpe ratio for measuring risk-adjusted performance of a dynamic portfolio. We compute optimal strategies for a portfolio investment problem motivated by the German DAX 30 Index and we evaluate and analyze the dependence of the $CVaRD$-based Sharpe ratio on the utility function and the associated risk aversion level.